Skip to content

[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models#24982

Merged
tlrmchlsmth merged 39 commits intovllm-project:mainfrom
tlrmchlsmth:tp_attn_fix_more_models
Sep 27, 2025
Merged

[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models#24982
tlrmchlsmth merged 39 commits intovllm-project:mainfrom
tlrmchlsmth:tp_attn_fix_more_models

Conversation

@tlrmchlsmth
Copy link
Copy Markdown
Member

@tlrmchlsmth tlrmchlsmth commented Sep 16, 2025 •
edited by github-actions bot
Loading

Purpose

Prior to this PR, in many cases, using TP Attn and EP MoEs with --tensor-parallel-size N --data-parallel-size M --enable-expert-parallel would result in factor N redundant work in the MoE layers.

This PR extends #24134 to other models, and to the naive and allgather_reducescatter All2All backends.

Test Plan

vllm serve {{MODEL}} -tp 2 -dp 2 --enable-expert-parallel --port 8192

lm_eval --model local-completions --tasks gsm8k --model_args model={{MODEL}},base_url={{BASE_URL}}/v1/completions,num_concurrent=50,max_retries=3,tokenized_requests=False --limit 100

Test Result

Qwen/Qwen3-30B-A3B-FP8:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.88|±  |0.0327|
|     |       |strict-match    |     5|exact_match|↑  | 0.94|±  |0.0239|

Qwen/Qwen3-Next-80B-A3B-Instruct (with --enforce-eager due to #25437):

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.80|±  |0.0402|
|     |       |strict-match    |     5|exact_match|↑  | 0.74|±  |0.0441|

meta-llama/Llama-4-Scout-17B-16E:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.82|±  |0.0386|
|     |       |strict-match    |     5|exact_match|↑  | 0.82|±  |0.0386|

ibm-granite/granite-4.0-tiny-preview (with --enforce-eager due to #25437 (comment)):

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.58|±  |0.0496|
|     |       |strict-match    |     5|exact_match|↑  | 0.55|±  |0.0500|

openai/gpt-oss-20b (main at TP4 is almost the same):

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3685|±  |0.0133|
|     |       |strict-match    |     5|exact_match|↑  |0.2365|±  |0.0117|
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@tlrmchlsmth
Apply TP Attn + EP MoE fix to other models
b83e888 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify mergify bot added deepseek Related to DeepSeek models qwen Related to Qwen models labels Sep 16, 2025
@tlrmchlsmth
llama4
1ae573a 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
llama4 eagle
9b969ee 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 17, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tlrmchlsmth.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 17, 2025
@tlrmchlsmth tlrmchlsmth added this to the v0.11.0 milestone Sep 18, 2025
@tlrmchlsmth
Qwen3-Next
d3bc2cf 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
Merge branch 'main' into tp_attn_fix_more_models
f1f3f63 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
Use SP for AG RS All2All backend
2ba92e7 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
WIP debugging
84d57d3 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
WIP implementing for naive and ag_rs a2a
42b9a0c 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify mergify bot added llama Related to Llama models speculative-decoding labels Sep 21, 2025
@tlrmchlsmth
Merge branch 'main' into tp_attn_fix_more_models
9cff96c 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify mergify bot removed the needs-rebase label Sep 21, 2025
@tlrmchlsmth
WIP making sure it works when the model isn't SP.
d5daf2c 
Runs but wrong answer in this case

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
Fixes and cleanup
c3d7c76 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth tlrmchlsmth marked this pull request as ready for review September 22, 2025 15:10
@wuxun-zhang wuxun-zhang mentioned this pull request Sep 28, 2025
Merged
@MengqingCao MengqingCao mentioned this pull request Sep 28, 2025
Open
xuechendi pushed a commit to vllm-project/vllm-gaudi that referenced this pull request Sep 30, 2025
@wuxun-zhang
Support sequence parallel MOE after upstream #24982 (#285)
922a18f 
After vllm-project/vllm#24982 merged, sequence
parallel MOE will be turned on when `enable_expert_parallel=True`,
`tp_size > 1` and `dp_size > 1`. Since for Gaudi, there is no choice for
`VLLM_ALL2ALL_BACKEND`, we can not easily bypass it. So this PR aims to
support the feature.

```python
class ParallelConfig:

  @Property
    def use_sequence_parallel_moe(self) -> bool:
        return (envs.VLLM_ALL2ALL_BACKEND
                in ("allgather_reducescatter", "naive",
                    "deepep_high_throughput", "deepep_low_latency")
                and self.enable_expert_parallel
                and self.tensor_parallel_size > 1
                and self.data_parallel_size > 1)

```

Update:
No hard requirement on vllm-project/vllm#25828

---------

Signed-off-by: Wuxun Zhang <wuxun.zhang@intel.com>
@ilmarkov ilmarkov mentioned this pull request Oct 1, 2025
Closed
5 tasks
iboiko-habana pushed a commit to iboiko-habana/vllm-gaudi that referenced this pull request Oct 2, 2025
@wuxun-zhang @iboiko-habana
Support sequence parallel MOE after upstream #24982 (vllm-project#285)
3eedb54 
After vllm-project/vllm#24982 merged, sequence
parallel MOE will be turned on when `enable_expert_parallel=True`,
`tp_size > 1` and `dp_size > 1`. Since for Gaudi, there is no choice for
`VLLM_ALL2ALL_BACKEND`, we can not easily bypass it. So this PR aims to
support the feature.

```python
class ParallelConfig:

  @Property
    def use_sequence_parallel_moe(self) -> bool:
        return (envs.VLLM_ALL2ALL_BACKEND
                in ("allgather_reducescatter", "naive",
                    "deepep_high_throughput", "deepep_low_latency")
                and self.enable_expert_parallel
                and self.tensor_parallel_size > 1
                and self.data_parallel_size > 1)

```

Update:
No hard requirement on vllm-project/vllm#25828

---------

Signed-off-by: Wuxun Zhang <wuxun.zhang@intel.com>
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025
@tlrmchlsmth @pdasigi
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (vllm-pro…
b106db1 
…ject#24982)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025
@ywang96 @pdasigi
[Bugfix] Fix Qwen3-VL regression from vllm-project#24982 (vllm-projec…
a0ccd47 
…t#25814)

Signed-off-by: Roger Wang <hey@rogerw.io>
yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@tlrmchlsmth @yewentao256
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (#24982)
e94aabe 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@ywang96 @yewentao256
[Bugfix] Fix Qwen3-VL regression from #24982 (#25814)
495f368 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
@jmkuebler jmkuebler mentioned this pull request Oct 8, 2025
Closed
1 task
choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request Oct 11, 2025
@tlrmchlsmth @choprahetarth
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (vllm-pro…
b5ef649 
…ject#24982)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request Oct 11, 2025
@ywang96 @choprahetarth
[Bugfix] Fix Qwen3-VL regression from vllm-project#24982 (vllm-projec…
7d68d87 
…t#25814)

Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: simon-mo <simon.mo@hey.com>
shyeh25 pushed a commit to shyeh25/vllm that referenced this pull request Oct 14, 2025
@tlrmchlsmth @shyeh25
Apply TP Attn + EP MoE fix to other models (vllm-project#24982)
786d867 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
shyeh25 pushed a commit to shyeh25/vllm that referenced this pull request Oct 14, 2025
@ywang96 @shyeh25
Fix Qwen3-VL regression from vllm-project#24982 (vllm-project#25814)
3809fae 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: simon-mo <simon.mo@hey.com>
@tlrmchlsmth tlrmchlsmth mentioned this pull request Oct 15, 2025
Merged
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@tlrmchlsmth @lywa1998
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (vllm-pro…
22c55b8 
…ject#24982)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@ywang96 @lywa1998
[Bugfix] Fix Qwen3-VL regression from vllm-project#24982 (vllm-projec…
85f7324 
…t#25814)

Signed-off-by: Roger Wang <hey@rogerw.io>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@tlrmchlsmth @alhridoy
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (vllm-pro…
7640427 
…ject#24982)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@ywang96 @alhridoy
[Bugfix] Fix Qwen3-VL regression from vllm-project#24982 (vllm-projec…
10061b7 
…t#25814)

Signed-off-by: Roger Wang <hey@rogerw.io>
@tlrmchlsmth tlrmchlsmth mentioned this pull request Oct 30, 2025
Merged
5 tasks
@frost-intel frost-intel mentioned this pull request Nov 4, 2025
Merged
5 tasks
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
@tlrmchlsmth @rtourgeman
[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (vllm-pro…
9ee5e00 
…ject#24982)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@ProExpertProg ProExpertProg mentioned this pull request Nov 21, 2025
Open
1 task
@AlfredYyong AlfredYyong mentioned this pull request Nov 28, 2025
Closed
1 task
@wxsIcey wxsIcey mentioned this pull request Dec 10, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ProExpertProg ProExpertProg ProExpertProg approved these changes

@youkaichao youkaichao youkaichao approved these changes

@sighingnow sighingnow Awaiting requested review from sighingnow sighingnow is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337

@ywang96 ywang96 Awaiting requested review from ywang96

Assignees

No one assigned

Labels

ci/build deepseek Related to DeepSeek models gpt-oss Related to GPT-OSS models llama Related to Llama models multi-modality Related to multi-modality (#4194) qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding

Projects

gpt-oss Issues & Enhancements
Status: Done

Milestone

v0.11.0 Cherry Picks

Development

Successfully merging this pull request may close these issues.

4 participants

@tlrmchlsmth @ProExpertProg @youkaichao @simon-mo